Improving Ranking by Respecting the Multidimensionality and Uncertainty of User Preferences

Rankings or ratings are popular methods for structuring large information sets in search engines, e-Commerce, e-Learning, etc. But do they produce the right rankings for their users? In this paper, we give an overview of major evaluation approaches for rankings as well as major challenges facing the use and usability of rankings. We point out the importance of an interdisciplinary perspective for a truly user-centric evaluation of rankings. We then focus on two central problems: the multidimensionality of the criteria that influence both users’ and systems’ rankings, and the randomness inherent in users’ preferences. We propose multicriteria decision analysis and the integration of randomness into rankings as solution approaches to these problems. We close with an outlook on new challenges arising for ranking when systems address not only individuals, but also groups.